Skip to content

Conversation

spairy
Copy link

@spairy spairy commented Oct 20, 2025

Which issue(s) this PR fixes:
None.

What this PR does / why we need it:
Supports parsing array of basic data types when analyzing YAML configuration files.

The current behavior of Fluentd:

  • Support YAML Array format for $args only.
  • Other Array options must be specified in a comma-separated format.
    • retryable_response_codes: 503, 504
    • If there’s only a single value, it must be specified as String or with the trailing comma.
      • retryable_response_codes: "503"
      • retryable_response_codes: 503,

This supports YAML Array format for all options.

retryable_response_codes:
  - 503
retryable_response_codes: [503]

Note: This PR doesn't address the issue where setting an Int directly to an Array option causes ConfigError: retryable_response_codes: 503 (The C case in #5126 (comment))
It could be a different issue.

Docs Changes: Not Need

Release Note: Same as the title.

@daipom daipom changed the title Supports parsing of basic data types when analyzing YAML configuratio… Supports parsing of basic data types when analyzing YAML configuration files Oct 20, 2025
@daipom
Copy link
Contributor

daipom commented Oct 20, 2025

Thanks for this improvement!
Could you add DCO?
Could you explain this feature in more detail?

@spairy spairy changed the title Supports parsing of basic data types when analyzing YAML configuration files Supports parsing array of basic data types when analyzing YAML configuration files Oct 20, 2025
@spairy
Copy link
Author

spairy commented Oct 20, 2025

Gett error when config array of basic data types with fluentd.yaml, eg:

retryable_response_codes:
  - 503  

or
retryable_response_codes: [503]

This feature is to address this issue.

@daipom
Copy link
Contributor

daipom commented Oct 21, 2025

I see. There seems to be a bug.
Fluentd should be able to parse YAML Array format: https://docs.fluentd.org/configuration/config-file-yaml#multiline-support-for-quoted-string-array-and-hash-values

I have confirmed the following behaviors.

  • A: Can parse Array format for $tag.
  • B: Cannot parse Array format for normal Array options.
  • C: Cannot parse a single Int for normal Array options.
    • It appears that the workaround is to specify a single String. (See A example below)

A: Can parse Array format for $tag

config:
  - source:
      $type: http
  - match:
      $type: stdout
      $tag: server.**
      format:
        $type: csv
        fields: message
      buffer:
        $arg:
          - tag
          - time
        $type: memory
        timekey: 1h
        flush_mode: immediate
  - source:
      $type: sample
      tag: client
  - match:
      $type: http
      $tag: client.**
      endpoint: http://localhost:9880/server
      retryable_response_codes: "503"
      buffer:
        flush_mode: immediate

B: Cannot parse Array format for normal Array options.

fluentd/lib/fluent/config/yaml_parser/parser.rb:145:in `section_build': undefined method `each' for "message":String (NoMethodError)
config:
  - source:
      $type: http
  - match:
      $type: stdout
      $tag: server.**
      format:
        $type: csv
        fields:
          - message
      buffer:
        $arg:
          - tag
          - time
        $type: memory
        timekey: 1h
        flush_mode: immediate
  - source:
      $type: sample
      tag: client
  - match:
      $type: http
      $tag: client.**
      endpoint: http://localhost:9880/server
      retryable_response_codes:
        - 503
      buffer:
        flush_mode: immediate

C: Cannot parse a single Int for normal Array options.

[error]: config error file="..." error_class=Fluent::ConfigError error="array required but got 503"
config:
  - source:
      $type: http
  - match:
      $type: stdout
      $tag: server.**
      format:
        $type: csv
        fields: message
      buffer:
        $arg:
          - tag
          - time
        $type: memory
        timekey: 1h
        flush_mode: immediate
  - source:
      $type: sample
      tag: client
  - match:
      $type: http
      $tag: client.**
      endpoint: http://localhost:9880/server
      retryable_response_codes: 503
      buffer:
        flush_mode: immediate

@daipom daipom self-requested a review October 21, 2025 00:43
@daipom
Copy link
Contributor

daipom commented Oct 21, 2025

Hmm, it looks like the current implementation intentionally treats only $args as an Array.
Is this a documentation mistake, or was something missed in the implementation?

config.each do |key, val|
if val.is_a?(Array)
val.each do |v|
sb.add_section(section_build(key, v, indent: indent + @base_indent))
end
elsif val.is_a?(Hash)
harg = val.delete('$arg')
if harg.is_a?(Array)
# To prevent to generate invalid configuration for arg.
# "arg" should be String object and concatenated by ","
# when two or more objects are specified there.
sb.add_section(section_build(key, val, indent: indent + @base_indent, arg: harg&.join(',')))
else
sb.add_section(section_build(key, val, indent: indent + @base_indent, arg: harg))
end
else
sb.add_line(key, val)
end
end

@Watson1978 Watson1978 self-requested a review October 21, 2025 01:19
@daipom
Copy link
Contributor

daipom commented Oct 21, 2025

It’s still unclear whether the current implementation is intentional or an oversight (possibly a bug).
Here’s the current behavior of Fluentd.

  • Support YAML Array format for $args only.
  • Other Array options must be specified in a comma-separated format.
    • retryable_response_codes: 503, 504
    • If there’s only a single value, it must be specified as String or with the trailing comma.
      • retryable_response_codes: "503"
      • retryable_response_codes: 503,

@daipom
Copy link
Contributor

daipom commented Oct 21, 2025

We supported YAML in the following:

There hasn’t been any particular discussion about handling Arrays. There are no test cases for it, but it’s clearly documented.

@daipom
Copy link
Contributor

daipom commented Oct 21, 2025

The following code is to parse the multiple sections.

if val.is_a?(Array)
val.each do |v|
sb.add_section(section_build(key, v, indent: indent + @base_indent))
end

@daipom
Copy link
Contributor

daipom commented Oct 21, 2025

There are two cases for Arrays: one where they should be recognized as sections, and another where they should be treated as single options.
Currently, the internal logic forces them to be recognized as sections.
To support Array options, we need to add cases like in this PR.

Copy link
Contributor

@daipom daipom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
The direction of this fix looks good!

Since the current implementation assumes either an Array or a Hash as a section, how about checking it like this?

@daipom
Copy link
Contributor

daipom commented Oct 21, 2025

I’d appreciate it if you could add some test cases.
Please let me know if that’s difficult.

sub_test_case "yaml config" do

@daipom
Copy link
Contributor

daipom commented Oct 21, 2025

@daipom daipom added this to the v1.20.0 milestone Oct 21, 2025
@daipom
Copy link
Contributor

daipom commented Oct 21, 2025

Whether to backport this needs to be discussed.

@spairy spairy force-pushed the parse_yaml_for_bug_of_array branch from efe72e6 to c160e7d Compare October 21, 2025 05:40
spairy and others added 3 commits October 21, 2025 13:49
Co-authored-by: Daijiro Fukuda <[email protected]>
Signed-off-by: Hear_Y <[email protected]>
Co-authored-by: Daijiro Fukuda <[email protected]>
Signed-off-by: Hear_Y <[email protected]>

Supports parsing array of basic data types when analyzing YAML configuration files.

Signed-off-by: Hear_Y <[email protected]>
@spairy spairy force-pushed the parse_yaml_for_bug_of_array branch from c160e7d to 2264e5e Compare October 21, 2025 05:49
@spairy
Copy link
Author

spairy commented Oct 21, 2025

Thanks! Test code is difficult for me. I have add DCO. I have no other question. Please help to close this Commit if necessary.

@daipom
Copy link
Contributor

daipom commented Oct 21, 2025

Thanks!
OK! We will add some test codes to this PR.

Signed-off-by: Daijiro Fukuda <[email protected]>
Copy link
Contributor

@daipom daipom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! LGTM.

@daipom daipom changed the title Supports parsing array of basic data types when analyzing YAML configuration files YAML config: Supports parsing array format Oct 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants